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BACKGROUND 

9 A number of local National Weather Service (NWS) forecast offices run a 
version of the Weather Research and Forecasting (WRF) numerical 
weather prediction (NWP) model called the Environmental Modeling 
System (EMS) at high resolution to complement global and larger-scale 
regional model guidance 

9 While forecasters may be able to eventually understand the biases in 
their model through daily use, effectively evaluating model performance 
requires a combination of quantitative metrics and case studies 

9 The Model Evaluation Tools (MET) software package was developed at 
National Center for Atmospheric Research (NCAR) Developmental 
Testbed Center (DTC) as a highly-configurable, state-of-the-art suite of 
verification tools for comparing gridded model output to point-based 
observations or gridded observations 

9 The DTC manages and distributes the MET code, but it is up to individual 
users to configure namelists and properly link model forecasts and 
verifying observations, which can take considerable time for those not 
familiar with how the software works 

9 The objective of this project is to develop and disseminate a set of 
dynamic scripts that enable researchers and operational weather 
forecasters to quickly evaluate their model output without needing 
detailed working knowledge of MET 

9 The MET Scripts make it possible to process statistics for multiple 
forecasts initialized at the same date and time (e.g., a control and 
experiment, or model sensitivity studies) 


APPLICATIONS 

9 The SPoRT MET Scripts have been used by a number of collaborators to easily implement objective verification of operational local model runs; details 
of select projects are outlined below 


1. CONVECTIVE INITIATION (SEE WAF/NWP 14.6) 

9 Collaboration with NWS Southern Region to 
investigate how well local models forecast 
convective initiation (Cl) 

9 Improvements to Cl found with inclusion of 
SPoRT higher-resolution land and water surface 
datasets, but model tended to be too dry 


9 Currently using 
MET Scripts to 
evaluate optimal 
physics 
configuration 
within WRF to 
simulate Cl using 
new higher- 
resolution land 
and water surface 
datasets (see 
sample results 
in Fig. 2) 

9 Final results to 
provide guidance 
to local modelers 
in the SEUS 
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Figure 2. Heidke Skill Score (HSS) for 3-hr 
precipitation at 10-mm thresholds for a set of 
EMS forecasts using (a) high-resolution SPoRT 
data and (b) standard initialization fields. 


2. LAKE EFFECT SNOW 

9 NWS Eastern Region runs an operational local 
model ensemble to forecast the onset, 
dissipation, and inland penetration of lake effect 
snow 

9 Some members of their ensemble use SPoRT 
high-resolution (2-km) lake surface temperature 
(LST) instead of the legacy RTG LST (1/12-deg) 

9 MET scripts is being used to evaluate the 
results of individual ensemble members 



FCST_HR 

■ ERSTRCNTL m ERSTRSPORT 

Figure 3. Sample output from Perl graphing module showing 6-hr 
accumulated precipitation HSS for two individual ensemble members: 
one using standard initialization (EASTRCNTL; black) and one using 
SPoRT initialization (EASTRSPORT; red) 


3. MESOAMERICA/KENYA (SEE WAF/NWP 39.4) 

9 Collaboration with NASA/SERVIR to enhance 
modeling capabilities at hubs 

9 EMS for Mesoamerica is being run in the cloud; 
EMS for Kenya being run locally at SPoRT 

9 SPoRT is running MET in near real-time 
(constrained by observation data latency) to 
verify the operational model output and posting 
stat figures on website for evaluation 

9 Hope to transition scripts to both hubs by the 
end of 2014 
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Figure 4. Sample output from Perl graphing module showing 2-m 
temperature mean error (i.e. bias) for a control (KENCNTL; black) and 
experiment (KENLIS; red) over an East Africa domain 


CODE INFRASTRUCTURE 


namelist.met 

• User configures options for verification, 
including initialization date(s)/time(s), 
variables, stat types, nad precipitation 
thresholds & accumulations 

• Read into each Perl program to generate 
configuration files for each part of MET 


• Open source scripts written in Perl with included modules to run various components of MET (see red stars in Fig. 
1) that wrap around the individual MET executables allowing for “set it and forget it” statistics processing 

• Current version of SPoRT MET Scripts (v4.1.2) is compatible with latest version of MET (v4.1 ; including bug fixes) 

• Unique features: 

• Single configuration file for all MET programs * Configure Stage IV precipitation for different accumulations 

• Download and configure MADIS observations 9 Generate hourly quick-look plots for all statistics 


runPointStat.pl runGridStat.pl 

runPointStatAnalysis.pl runGridStatAnalysis.pl 


runSPoRTMETcripts.pl 

Program that is executed to run script 
package 

By setting logical variables in namelist 
users can select only the parts of MET 
they want to run 



obtainObservations.pl 

Download Stage IV 
precipitation analyses 
Download and create 
ASCII files from MADIS 
Download PrepBufr 
files (for OCONUS 
applications) 


makePlots.pl 

Optional open 
source plotting 
scripts to 
visualize stats 
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Figure 1. Schematic of SPoRT MET Scripts with blue boxes indicating additional capabilities provided by the scripts. Center figure is schematic of MET (red stars indicate current functionaly of scripts) from the MET User’s Guide. 


HOW DO I GET THE SCRIPTS? 

9 ftp://geo.nsstc.nasa.gov/SPoRT/modeling/metScripts or 
scan the QR code below 

9 User must first download and compile MET source code 
from DTC (http://www.dtcenter.org) 

9 If you plan to use the scripts for your research please 
acknowledge NASA SPoRT as the script developers 

9 Contact Brad Zavodsky at brad.zavodsky@nasa.gov for 
more information or if you have questions about the 
scripts and/or their use 



FUTURE DEVELOPMENTS 

9 Update the scripts as new versions of MET are released 

9 Add capabilities to perform object-based verification 
(MODE) for precipitation and other gridded fields 

9 Integrate MET Scripts into EMS and output to AWIPS II 

9 Add verification of NASA satellite datasets 













